Skip to content

refactor(queue/mysql): immutable log, delivery state, heartbeat store, fair leasing#131

Merged
behinddwalls merged 1 commit into
mainfrom
preetam/queue
Mar 10, 2026
Merged

refactor(queue/mysql): immutable log, delivery state, heartbeat store, fair leasing#131
behinddwalls merged 1 commit into
mainfrom
preetam/queue

Conversation

@behinddwalls
Copy link
Copy Markdown
Collaborator

@behinddwalls behinddwalls commented Mar 9, 2026 •
edited
Loading

Summary

  • Add delivery state store for tracking per-consumer message state separately from the immutable log
  • Add subscriber heartbeat store for liveness detection and fair partition leasing
  • Refactor message store to append-only immutable log with contiguous watermark tracking
  • Refactor partition lease store with heartbeat-based fair leasing
  • Add batch query support for offset and delivery state operations
  • Add GC throttling and dead code removal
  • Fix make fmt to skip read-only pkg/mod directory

Test Plan

  • Unit tests for all new and refactored stores
  • Integration tests cover full subscriber lifecycle

Issues

Stack

  1. @ refactor(queue/mysql): immutable log, delivery state, heartbeat store, fair leasing #131
  2. docs(queue/mysql): update RFC and README for immutable log architecture #130
  3. refactor(queue/mysql): event-driven integration tests, zero sleeps #132

@behinddwalls behinddwalls requested review from a team and sbalabanov as code owners March 9, 2026 02:42
@behinddwalls behinddwalls mentioned this pull request Mar 9, 2026
Merged
@behinddwalls
Copy link
Copy Markdown
Collaborator Author

behinddwalls commented Mar 9, 2026 •
edited
Loading

@sbalabanov somehow this PR: #126 got messed up, so created this one to address comments, it's larger refactor to fix double logging, double metrcis, and consolidate some duplicated code and handle error swallowing at the callsite.

PR for fixing test sleep/wait: #132

This was referenced Mar 9, 2026
Merged
Merged
sbalabanov
sbalabanov approved these changes Mar 9, 2026
View reviewed changes
Comment thread extension/queue/mysql/schema/queue_subscriber_heartbeats.sql
Comment thread extension/queue/mysql/delivery_state_store.go
Comment thread extension/queue/mysql/delivery_state_store.go
Comment thread extension/queue/mysql/delivery_state_store.go Outdated
Comment thread extension/queue/mysql/errors.go
Comment thread extension/queue/mysql/subscriber.go Outdated
Comment thread extension/queue/mysql/subscriber.go
Comment thread extension/queue/mysql/subscriber_heartbeat_store.go Outdated
Comment thread test/integration/extension/queue/mysql/queue_test.go
Comment thread test/integration/extension/queue/mysql/queue_test.go
@behinddwalls
refactor(queue/mysql): immutable log, delivery state, heartbeat store…
45eec0a 
…, fair leasing

- Add delivery state store for tracking per-consumer message state separately from the immutable log
- Add subscriber heartbeat store for liveness detection and fair partition leasing
- Refactor message store to append-only immutable log with contiguous watermark tracking
- Refactor partition lease store with heartbeat-based fair leasing
- Add batch query support for offset and delivery state operations
- Add GC throttling and dead code removal
- Fix make fmt to skip read-only pkg/mod directory
@behinddwalls behinddwalls added this pull request to the merge queue Mar 10, 2026
Merged via the queue into main with commit 1b8d967 Mar 10, 2026
11 checks passed
@github-actions github-actions Bot deleted the preetam/queue branch March 10, 2026 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sbalabanov sbalabanov sbalabanov approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@behinddwalls @sbalabanov